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SECTION  ONE 


SYSTEMS  ENGINEERING  MODEL  BACKGROUND 


1.1  Best  Manufacturing  Practices  (BMP)  Program 

The  Best  Manufacturing  Practices  (BMP)  Program  was  established  by  the  Navy  in  1985  with  the 
mission  to  collect,  analyze  and  disseminate  information  on  proven  techniques  used  by  the  government  and 
industry  throughout  the  acquisition,  development  and  field  support  process.  Typically,  this  information  is 
obtained  via  on-site  surveys  of  a  company’s  production  lines  and  through  meetings  with  development  and 
production  managers  and  personnel.  The  BMP  Program  has  four  main  products:  Best  Practices  Survey 
Reports,  the  Program  Manager’s  Workstation  (PMWS),  WEB  and  Virtual  Office  services,  and  Systems 
Engineering  work  for  Program  Managers. 

SURVEY  REPORTS:  After  completion  of  a  BMP  survey,  a  report  is  developed  which  identifies  best 
practices  used  by  that  company  in  design,  test,  manufacturing,  and  support  of  their  products.  The  survey 
report  is  then  distributed  throughout  the  BMP  user  community.  In  addition,  potential  industry-wide  problems 
are  also  documented  (with  the  company’s  permission)  and  shared  in  an  effort  to  encourage  cooperation  in 
solving  these  problems.  One  measure  of  the  success  of  this  program  is  the  quantity  and  quality  of  design  and 
manufacturing  information  shared  between  companies. 

PMWS:  After  several  years  of  successful  BMP  Program  operation,  it  was  determined  that  an 
improved  and  automated  methodology  was  needed  to  collect,  analyze  and  disseminate  program  and 
engineering  process  information.  From  this  vision,  the  concept  of  a  ’’Workstation”  was  pursued;  and,  since 
the  information  that  was  to  be  made  available  would  be  extremely  valuable  to  Program  Managers,  the  system 
was  named  Program  Manager’s  Workstation  (PMWS). 

WEB  SERVICES:  The  BMPCOE  has  a  very  capable  web  services  group  specializing  in  large 
database  and  back  office  support  using  CORBA,  JAVA,  and  other  effective  technologies.  The  web-based 
virtual  office,  one  of  our  top  products,  enables  teams  to  work  together  from  any  web  connection.  These 
Virtual  Offices  are  fully  encrypted  and  utilize  a  self-managing  architecture  to  significantly  reduce  costs. 

SYSTEMS  ENGINEERING:  Our  systems  engineering  support  includs  detailed  metallurgical  and 
material  properties  studies;  hardware  failure  analysis;  pedigree  program  development;  and  risk  management. 
The  Systems  Engineering  model  is  used  heavily  by  this  group  in  its  support  to  Program  Managers.  The 
Systems  Engineering  Model  represents  the  best  of  DoD  and  commercial  practices,  and  is  proving  successful 
on  military  and  commercial  programs  alike.  Central  to  the  PMWS  expert  systems,  the  model  consists  of  a 
flowchart  and  supporting  narratives  that  guide  the  user  through  all  the  critical  technical  processes.  The 
flowchart  shows  key  activities  in  the  engineering  process,  as  well  as  principal  tools  and  guidance  which  the 
PMWS  provides  to  assist  the  user.  The  narratives  provide  amplifying  information  on  the  flowchart’s 
engineering  process  activities. 


1.2  PMWS  Philosophy  -  Workload  Reduction 

The  Program  Manager’s  Workstation  (PMWS)  is  the  pseudonym  for  a  series  of  interrelated  software 
environments  and  knowledge  based  packages  designed  to  provide  timely  acquisition  and  engineering 
information  to  the  user.  Workload  reduction  is  a  top  priority  of  the  PMWS. 
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Typical  project  management  programs,  based  on  cost  and  schedule,  used  the  graphical  power  of  the 
computer  to  show  the  user  numerous  items  on  the  screen  at  once.  The  power  of  these  programs  to  display  and 
’’scroll”  through  data  is  enormous,  and  the  graphical  interface  makes  it  very  easy  for  the  user  to  manipulate 
the  data.  However,  these  programs  do  little  to  reduce  the  workload  on  an  already  overworked 
manager/engineer.  In  fact,  studies  have  shown  that  displaying  large  amounts  of  data  tends  to  confiise  the  user 
instead  of  focusing  on  the  critical  items  needing  his  attention.  Rather  than  display  large  amounts  of  data,  or 
even  the  entire  critical  path,  the  PMWS  typically  shows  the  user  the  one  to  five  most  critical  items  that  should 
be  addressed  NOW! 


1.3  Unique  PMWS  Capabilities 

The  PMWS  is  unique  among  existing  program  management  "tools”  and  is  meant  to  fill  a  long  standing 
deficiency.  Other  tools  (e.g.,  Microsoft  Project)  center  mainly  on  cost  and  schedule  indicators/relationships 
(typically,  CDRL,  WBS,  or  DoD  5000  oriented)  and  attempt  to  manage  an  acquisition  based  on  their  current 
(and  sometimes  projected)  status.  There  are  many  good  programs  of  this  type  on  the  market  today  and  the 
PMWS  is  not  attempting  to  duplicate  their  functions.  While  these  tools  do  facilitate  organizational  plans  and 
coordination,  despite  their  use  we  continue  to  find  acquisition  programs  failing  during  development  and  the 
transition  to  production. 


1.4  Focus  on  Engineering  Process 

PMWS  tools  are  centered  on  the  engineering  process  itself,  and  therefore,  are  very  process  oriented.  If 
all  engineering  processes  are  understood,  proper,  and  under  control,  the  results  will  be  as  good  as  the  state-of- 
the-art  will  allow. 

Cost  and  schedule  are  downstream  indicators  of  technical  problems  (sometimes  far  downstream). 

This  is  why  PMWS  tools  manage  technical  and  process  risks,  so  that  engineering  problems  are  surfaced  and 
given  visibility  at  the  earliest  possible  point.  In  this  way  they  can  be  addressed  and  mitigated  before  cost  and 
schedule  problems  are  indicated. 

For  example,  if  the  risk  management  (TRIMS)  portion  of  the  PMWS  indicates  that  your  program  has/is 
incurring  a  risk  due  to  a  Design  Reference  Mission  Profile  not  being  conducted;  the  user  can  go  directly  to  the 
knowledge  base  (i.e.,  KnowHow  and  BMP  Database)  and  get  FULL  details  on  why  a  Design  Reference 
Mission  Profile  is  needed  and  how  to  develop  one,  including  metrics  and  examples.  The  PMWS  will  identify 
such  problems  early  in  the  design  process.  However,  non-technical  (e.g.,  cost,  schedule)  tools  will  not  expose 
the  problem  until  well  into  the  operational  test  and  evaluation  process.  This  is  a  typical  example  of  how  the 
PMWS  can  work  for  you. 

The  PMWS  is  a  dynamic  new  tool  that  will: 

(1)  give  you  rapid  access  to  the  information  you  need  to  manage  your  program 

(2)  provide  technical  risk  assessment  for  your  specific  program  requirements 

(3)  show  you  successful  industry  solutions  to  reduce  technical  risk  and  improve  quality  and 
productivity. 

The  PMWS  is  a  series  of  expert  systems  and  decision  assistance  tools.  It  provides  knowledge,  insight, 
experience,  and  communication  as  an  extensive  network  of  information  and  software  resources  which  are 
easily  accessible  through  your  PC. 
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1.5  Technology  Transfer  Nationwide 


The  automated,  computerized,  commimication  link  connecting  all  BMP  Program  participants,  including 
the  network  of  PMWS  users,  is  called,  appropriately  enough,  BMPnet.  Currently,  BMPnet  is  used  to  facilitate 
communication  between  DoD  components.  Department  of  Commerce  (DOC),  Department  of  Energy  (DOE), 
Federal  Aviation  Administration  (FAA),  DoD  Centers  of  Excellence,  and  commercial  companies.  Program 
Managers  can  now  communicate  with  each  other  and  gain  from  each  other’s  experience  simply  and  easily 
when  solving  problems. 

Other  activities  of  the  BMP  Program  include  developing  and  publishing  technical  manuals  and 
guideline  documents,  producing  instructional  videos,  and  working  with  the  Navy/DoD  Technical  Centers  of 
Excellence. 

During  the  development  of  the  PMWS,  several  BMPnet  prototype  testbeds  were  established  to  evaluate 
different  system  implementation  concepts  and  designs.  The  one  design,  which  we  named  BMPnet,  proved 
very  flexible  and  was  established  as  an  initial  BMP  system  host  during  that  study  phase. 


1.6  Systems  Engineering  Model 

The  Systems  Engineering  Model  was  developed  as  part  of  the  PMWS  to  improve  the  collection, 
analysis,  and  distribution  of  engineering  process  information.  The  Systems  Engineering  Model  includes 
of  a  flowchart  that  describes  key  engineering  process  activities,  as  well  as  their  locations,  interfaces,  and 
timing  with  respect  to  other  activities  in  the  acquisition  process.  In  addition,  the  flowchart  blocks  are  keyed  to 
assist  the  user  to  rapidly  access  a  supporting  narrative  description,  specific  electronic  tools,  and  related 
guidance  items  from  PMWS. 

Two  significant  additions  were  made  in  this  version  of  the  Systems  Engineering  Model.  They  are  the 
Technology  Maturation  Loop  and  the  Process  Maturation  Loop.  These  loops  describe  events  that  require 
repetitive  processes,  rendering  traditional  program  management  schedule  tools  ineffective.  Section  Four 
contains  additional  narrative  for  these  loops  (block  numbers  61  and  72  respectively)  as  well  as  a  separate 
discussion  in  Section  Five. 
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SECTION  TWO 


BEST  MANUFACTURING  PRACTICES 
CENTER  OF  EXCELLENCE  (BMPCOE) 
SYSTEMS  ENGINEERING  MODEL 


2.1  Acquisition  Reform 

The  Systems  Engineering  Model  is  a  direct  outgrowth  of  the  PMWS  expert  systems.  It  contains  the 
results  of  the  best  of  DoD  and  commercial  practices.  The  application  of  the  flowchart  contained  within  the 
model  is  proving  successftil  on  mihtary  and  commercial  programs  alike.  It  is  compliant  with  the  DoD  5000 
series,  NAVSO  P-6071,  DoD  4245.7-M,  and  NAVSO  P-3679.  It  is  invaluable  for  helping  individuals 
involved  at  any  point  in  the  acquisition  process,  including  the  application  of  Non-Developmental  Item  (NDI), 
now  that  they  are  not  omitting  any  critical  technical  processes. 

The  flowchart,  used  with  PMWS,  identifies  specific  PMWS  template  topics  much  as  an  electronic 
consultant.  Intelligent  search  routines  improve  access  to  automated  information  by  up  to  95%.  PMWS  is  a 
stand-alone  system  and  can  be  run  on  any  PC  or  network  under  Windows. 

The  flowchart  contains  blocks  that  show  key  engineering  process  activities.  These  blocks  are  in 
sequence  to  show  their  location,  interfaces,  and  timing  with  respect  to  other  activities  and  the  overall 
acquisition  process.  Under  the  blocks,  in  a  black  background,  are  additional  tools  and  guidance  items  that 
directly  support  that  block.  Information  on  these  and  other  related  items  is  easily  accessed  on  a  computer  by 
activating  the  powerful  search  option  at  the  bottom  of  the  computer  screen.  In  addition,  each  block  in  the 
flowchart  has  a  supporting  narrative  description  of  its  role  in  the  acquisition  process.  These  narratives  can  be 
selected  by  number  from  the  main  flowchart  menu  screen.  A  copy  of  the  flowchart  is  in  Section  3.0  and 
copies  of  the  narratives  are  contained  in  Section  4.0  of  this  document. 

2.2  Point  of  Contact  Information 

For  PMWS  or  BMPnet  technical  assistance,  call  the  help  desk  at  (301)  403-8179.  For  additional 
information  on  the  BMP  Program,  contact  the  BMP  Center  of  Excellence  at  (301)  403-8100. 
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SECTION  THREE 


BMPCOE 

SYSTEMS  ENGINEERING  MODEL  FLOWCHART 


3.1  Introduction 

This  section  is  intended  to  familiarize  the  user  with  the  Systems  Engineering  Model  Flowchart.  The 
flowchart  process  blocks  use  identification  numbers  on  the  LEFT  and  RIGHT  sides  of  the  process  blocks  to 
identify  initial  applications,  as  well  as  specific  critical  inputs.  The  numbers  on  the  LEFT  of  the  block  indicate 
the  initial  application  of  that  process.  The  numbers  on  the  RIGHT  of  the  block  indicate  a  previously 
referenced  process  with  specific  critical  inputs  needed  for  that  block.  In  addition  to  the  identification  numbers 
and  for  the  user’s  convenience,  PMWS  template  Name/Topics  are  identified  below  the  process  blocks.  These 
Name/Topics  can  be  found  in  the  PMWS  and  provide  specific  topics  and  tools  that  can  help  you  effectively 
conduct  the  applications. 

As  an  example,  Figure  1  on  page  3-2  has  been  extracted  from  the  actual  flowchart.  This  example  will 
familiarize  you  with  the  process  block  numbering  system  described  above. 
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The  following  narrative  describes  how  the  numbering  system  is  used  to  identify  process  blocks  in  the 
flowchart. 

•  Block  numbers  42  through  47  have  numbers  on  the  upper  LEFT.  That  indicates  this  is  the  first  time 
you  have  seen  this  activity,  and  introduces  a  new  process  application. 

•  Three  other  blocks  (Generate  Final  Spec  Tree,  Generate  Detail  Hardware  Level  Breakdown,  and 
Update  LCC/Logistics  Analysis)  have  no  numbers  on  the  upper  LEFT.  That  indicates  that  you  have 
previously  seen  this  activity. 

•  Block  44  (Draft  Software  User’s  Manual)  has  a  number  on  the  upper  LEFT,  indicating  this  is  the 
first  time  you  have  seen  this  activity.  This  block  also  has  numbers  on  the  upper  RIGHT,  indicating 
previously  referenced  processes  with  specific  critical  inputs  needed  to  conclude  this  activity. 

•  Five  blocks  (Perform  Human  Factors  Analysis,  Perform  ID/T  Analysis,  Generate  Final  Spec  Tree, 
Update  LCC/Logistics  Analysis,  and  Establish  Supplier  Requirements)  identify  specific  PMWS 
Template  topics  and  tools  in  obverse  (black  on  white  printing)  that  can  help  you. 


44> 
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(g) 


Generate 

Sojftware 
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Figure  1  -  Sample  Process  Block  Identification 


3.2  Flowchart 


The  following  pages  contain  the  complete  Systems  Engineering  Model  Flowchart. 
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Phase  0: 

Concept  Exploration 
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Phase  1: 

Program  Definition  &  Risk  Reduction  (Cont'd) 
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(Continuation  or  Conflj^ratlon  1Vfaua{{cnicnt  work) 
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Phase  2: 

Engineering  &  Manufacturing  Development 
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SECTION  FOUR 


FLOWCHART  SUPPORTING  NARRATIVES 


4.1  Narratives 

The  following  narratives  provide  additional  information  on  each  block  in  the  Systems  Engineering 
Model  Flowchart.  They  are  niunbered  as  they  appear  in  the  flowchart.  When  the  Systems  Engineering  Model 
is  installed  on  your  Personal  Computer  (PC),  you  can  access  the  narratives  simply  by  selecting  the 
corresponding  block  number  and  title  in  the  main  flowchart  menu. 


1.  Customer  Requirements  Definition 

These  three  areas  -  design  reference  mission  profile,  design  requirements,  and  funding  are  basic 
responsibilities  of  the  customer  and  must  be  derived  from  a  determination  of  mission  need.  Failure  to 
adequately  define  the  design  mission  profile,  adequately  state  design  requirements,  or  provide  sufficient  up 
front  funding  will  adversely  affect  the  program  for  its  entire  life  cycle. 


2.  Requirements  Hierarchy 

Requirements  are  not  all  equal  and  the  establishment  of  a  hierarchy  from  the  outset  is  critical  for  effective 
design,  trade,  and  cost  studies.  All  requirements  should  be  assigned  to  one  of  three  categories:  Mandatory, 
Highly  Desirable,  and  Optional. 


3.  Software  Development  Plan 

The  Software  Development  Plan  includes  the  development  and  testing  of  software.  The  software  test  plan  is 
generated  as  part  of  the  software  development  plan  and  is  completed  prior  to  the  start  of  coding.  Design  and 
test  disciplines  are  inputs  to  the  software  test  plan. 

The  software  test  planning  activity  includes  plans  for  the  building,  use,  and  maintenance  of  a  software 
simulator  and  produces  data  that  allows  completion  of  the  software  user’s  manual.  The  performance  of 
software  coding,  unit  test,  software  build,  and  software  and  software/hardware  integration  testing  is  described 
in  the  software  development  plan.  Software  development  is  integrated  with  prime  equipment  development  as 
practical. 


4.  System  Environmental  Profile 

MIL-SPEC  environments  are  reviewed,  analyzed,  and  tailored  to  describe  the  system's  production,  operation, 
and  support  environments  versus  time.  The  minimum  operating  and  non-operating  requirements  addressed 
are:  temperature,  vibration.  Electromagnetic  Interference/Electromagnetic  Pulse  (EMI/EMP),  Electrostatic 
Discharge  (ESD),  humidity,  altitude,  salt  spray/fog.  Nuclear,  Biological,  Chemical  (NBC),  sand/dust,  Foreign 
Object  Damage  (FOD),  and  production  contaminants.  Analyses  should  also  qualify  environmental  differences 
between  new  requirements  and  similarly  deployed  in-service  equipment. 
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5.  System  Functional  Profile 


Describes  prime  equipment  functional  requirements  (addressing  both  operational  and  production 
requirements)  versus  time  (both  mission  and  life  cycle).  First  or  Second  Level  Functional  flow  block 
diagrams  are  normally  developed  to  give  project  management  personnel  an  overall  functional  perspective. 


6.  Logistics  Profile 

Developed  to  represent  a  system’s  total  life  cycle  logistical  environment.  Special  attention  is  paid  to  both 
operator-related  and  maintenance-related  human  factors  and  maintainability  needs.  Addresses,  at  least: 

•  Definition  and  analysis  of  Operability  and  Supportability  (O&S),  e.g.,  available  manpower  skills, 
best  maintenance  concept,  trouble  report  system. 

•  Development  of  a  design  constraints/goals  list  following  O&S  analysis,  field  observations  and 
comparison  with  similarly  deployed  systems. 

•  Development  of  an  ILS  Plan. 

•  Development  of  a  Military  Manpower/Hardware  Integration  Program  (HARDMAN)  Plan. 


7.  Prime  Item  Development  Specifications  (PIDS),  Allocated  Baseline 

Converts  environmental,  logistical,  and  functional  profile  requirements  (including  in-house  production  needs) 
into  a  prime  item  development  specification.  Should  address  electrical,  mechanical,  and  software 
requirements;  operating  and  non-operating  environments,  support  constraints  and  production  needs.  These 
requirements  should  then  be  allocated  to  each  subassembly. 


8.  Supplier  Evaiuation/Market  Surveillance 

Suppliers  must  be  evaluated  in  terms  of  financial  stability  and  history  of  delivering  quality  products  and 
services.  The  market  must  be  surveyed  to  see  what  the  current  technical  base  has  “off-the-shelf’  and  what  it 
can  be  expected  to  develop  and  produce.  This  evaluation  should  identify  a  list  of  potential  suppliers  to 
engineering. 


9.  Non-Development  Item  (NDI) 

A  Non-Development  Item  covers  material  available  from  a  wide  variety  of  sources  with  little  or  no 
development  effort  required  by  the  Government.  NDIs  include: 

•  Any  item  of  supply  that  is  available  in  the  commercial  marketplace. 

•  Any  previously  developed  item  of  supply  that  is  in  use  by  a  department  or  agency  of  the  U.S.,  a  state 
or  local  government,  or  a  foreign  government  with  which  the  U.S.  has  a  mutual  defense  cooperation 
agreement. 
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•  Any  item  of  supply  described  above  that  requires  only  minor  modification  in  order  to  meet  the 
requirements  of  the  contracting  agency. 

•  Any  item  of  supply  that  is  currently  being  produced  that  does  not  meet  the  requirements  of  the  items 
described  above  solely  because  the  item  is  not  yet  in  use  or  is  not  yet  available  in  the  commercial 
marketplace. 

The  application  of  NDI  to  an  acquisition  should  be  viewed  as  a  matter  of  degree  rather  than  an  all  or  nothing 
proposition.  As  the  extent  of  NDI  utilized  in  an  acquisition  increases,  development  cost  and  time  are  reduced. 
Many  NDI  opportunities  require  modifying  an  existing  item  (e.g.,  ruggedize,  militarize)  and  incorporating 
NDIs  into  larger  elements  of  a  system. 

A  predominant  use  of  NDI  is  related  to  the  insertion  of  NDI  at  subsystem,  equipment,  component,  and  piece 
part  levels  in  major  developmental  programs.  These  opportunities  should  be  explored  as  part  of  system 
engineering  and  system  integration  processes.  The  use  of  NDI  in  satisfying  approved  operational 
requirements  could  involve  a  combination  of  items  (e.g.  commercial,  individual  or  joint  service,  foreign,  or 
standard  DoD)  coupled  with  and  integrated  into  a  significant  developmental  effort. 

The  major  benefits  of  NDI  acquisitions  are: 

•  Quick  response  to  operational  needs. 

•  Elimination  or  reduction  of  research  and  development  costs. 

•  Application  of  state-of-the-art  technology  to  current  requirements. 

•  Reduction  of  technical,  cost,  and  schedule  risks. 


10.  Design  Policy 

A  corporate  statement  supported  by  controlled  engineering  manuals,  procedures,  and  guidelines.  This  Design 
Policy  should  be  visible  and  followed  with  check  points  to  validate  compliance,  and  tailored  to  a  specific 
project  or  product  area. 


11.  Program  Design  Manual 

A  program  design  manual,  developed  and  introduced  to  the  design  team  before  the  start  of  design,  should 
address  such  issues  as  producibility  guidelines/standards,  parts  selection  and  derating,  testability  guidelines, 
and  other  design  requirements  for  compliance.  The  manual  should  be  approved  by  management,  and  should 
be  part  of  each  designer's  required  documentation.  Compliance  with  the  manual  is  verified  and  documented  at 
each  design  review. 


12.  Technology  Base  Survey 

An  important  early  study  is  of  the  capabilities  of  the  U.S.  industrial  bases  vers  product  requirements.  Will 
special  custom  tooling  need  to  be  made?  Who  has  done  this  before  and  with  what  variability.  Is  there 
adequate  availability  of  raw  materials.  How  mature  are  the  technologies  needed  to  build  your  product  and 
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how  many  single  source  suppliers  will  you  have?  These  and  other  similar  issues  need  to  be  resolved  and  used 
as  input  to  the  trade  studies  process. 


13.  Technical  Risk  Identiflcation  and  Mitigation  System  (TRIMS) 

TRIMS  is  one  of  the  PMWS  suite  of  tools.  TRIMS  currently  supports  risk  management  in  three  (3)  separate 
functional  areas:  Systems  Engineering,  Software  Development  and  Circuit  Testability.  In  addition  to  a 
technical  risk  management  system,  TRIMS  will  also  "rollup”  several  risk  assessments  into  higher  level 
reports.  It  is  also  an  excellent  action  item  tracking  system. 


14.  Corporate  Policies  and  Design  Policies 

Corporate  policies  and  procedures  are  the  heart  of  a  continuous  process  improvement  program;  unless  a 
process  is  baselined  and  tracked,  it  is  impossible  to  know  whether  or  not  you  are  improving  or  declining. 
Also,  without  clear  policies  and  procedures,  employees  do  not  know  what  is  expected  of  them,  or  how  to  get 
ahead. 

The  design  process,  supported  by  design  policies,  describes  all  the  actions  taken  that  culminate  in  a  set  of 
drawings  or  a  database  from  which  a  model  can  be  constructed  for  testing  to  verify  specification  compliance. 
Design  criteria  are  developed  and  proven  before  EMD,  and  may  be  updated  and  refined  as  experience  is 
gained  during  development.  Production  design  occurs  concurrently  with  the  other  elements  of  the  design 
process. 


15.  Personnel  Issues 

Despite  the  automation  in  factories  today,  people  still  design  and  assemble  the  lion’s  share  of  any  product. 
Companies  must  know  how  to  identify  and  retain  their  key  personnel,  as  well  as  allow  a  path  for  all  ideas  to 
flow  freely  upward  from  the  workforce.  Typical  management  channels  have  proven  ineffective  in  this  area. 


16.  Capital  Planning 

Capital  planning  is  critical  if  an  organization  is  to  remain  competitive.  Capturing  and  maintaining  a  business 
base  is  more  than  just  a  statement  on  a  business  plan  demanding  ”15%  growth.”  Markets  must  be  analyzed 
and  proper  investments  made  to  penetrate  or  maintain  your  business  base. 


17.  Specifications  Tree 

Based  on  PIDS,  a  complete  set  of  specifications  including  system,  hardware/software  interfaces, 
reliability/failure  rate,  testing,  and  functional  requirements  are  developed  down  to  the  Shop-Replaceable 
Assembly  (SRA)  level.  BIT  allocations/requirements  are  defined  at  each  level. 
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18.  Integrated  Test  Plan  (ITP) 


A  complete  ITP  is  developed  and  maintained,  addressing,  for  example,  all  development,  qualification,  and 
acceptance  testing  to  be  performed  by  the  company,  subcontractors,  and  the  government,  Development  Test 
and  Evaluation  (DT&E)  and  Operational  Test  and  Evaluation  (OT&E),  support  equipment/special  test 
equipment  evaluation,  and  acceptance  testing.  The  ITP  also  identifies  critical  test  resources  and  schedule 
constraints,  missing  and  redundant  test  efforts,  and  documents  use  of  similarity  of  design  to  justify  lack  of 
testing.  This  plan  requires  government  approval  and  calls  for  as  much  government  participation  in  testing  as 
is  practical. 


19.  Trade  Study  Plan 

A  broad  spectrum  of  trade  studies  is  initiated  during  concept  exploration.  These  trade  studies  continue  into 
engineering  and  manufacturing  development  as  a  logical  approach  to  selecting  the  best  design,  once  the 
mission  profile  and  design  requirements  have  been  specified.  The  final  selection  and  fine  tuning  of  the  design 
approach  must  consider  such  factors  as  producibility,  operational  suitability,  performance,  obsolescence,  cost, 
risk,  schedule,  alternatives,  and  technology  assessments. 


20.  Design  Surveys 

Surveys  include  Stress/Strength,  Worst  Case  Tolerance,  Thermal,  and  Vibration.  Surveys  are  scheduled, 
performed,  and  the  results  stored  and  checked  against  program  Design  Manual  criteria  to  ensure  compliance 
with  program  requirements.  Surveys  are  scheduled  to  allow  time  for  interpretation  and  proof  of  correction; 
surveys  are  redone  to  verify  corrective  action  or  when  there  is  a  significant  test  parameter  change. 


21.  Design  Analyses 

Required  analyses:  Stress/Strength,  Worst  Case  Tolerance,  Sneak  Circuit,  FMECA/FMEA,  Thermal, 
Logistics,  System  Safety,  and  Integrated  Diagnostics  &  Testability  (ID/T).  Others  (e.g.  Fault  Tree,  mass 
properties)  are  performed  as  needed.  All  results  are  documented  and  maintained  throughout  the  program’s 
life.  The  results  are  checked  against  the  program's  Design  Manual  criteria  to  ensure  compliance  with  program 
requirements.  Analyses  are  scheduled  and  performed  to  support  the  design  effort  and  agenda  of  design 
reviews.  Computer-Aided  Design/Computer- Aided  Engineering  tools  are  used  to  perform  analyses  whenever 
possible.  Design  updates  will  be  documented  using  CAD/CAE  when  possible. 


22.  Program  Plans 

Plans  (e.g.  safety,  reliability,  testability,  human  factors,  compatibility  assessment)  address  performance 
parameters  which  are  proven  during  System  Qualification  Testing.  Items  on  the  list  address  specific  system 
needs  and  may  show  up  as  deliverable  data  items.  Project  management  decides  which  plans  will  be  developed 
and  maintains  written  justification  for  excluding  any  plan.  Plans  are  released  and  government-approved  as 
part  of  the  Transition  Plan. 
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23.  Generate  Software  Functional  Baseline 


Formal  Configuration  Management  (CM),  which  includes  Software  Functional  Baselines  (FBLs),  is 
mandated  by  the  customer  after  customer-approved  baselines  are  established.  For  example,  formal  CM 
begins  at  contract  award  for  any  existing  system  specifications  called  out  in  the  contract.  However,  the  system 
specifications  may  not  always  be  provided  with  the  contract;  occasionally,  the  contractor  is  required  to 
develop  or  update  this  specification.  These  specifications  and  the  system-segment  specifications,  if  any, 
constitute  the  Functional  Baseline  (FBL). 

The  FBL  is  approved  documentation  describing  a  system’s  or  item’s  fiinctional  characteristics  and  the 
verification  required  to  demonstrate  the  achievement  of  those  specified  functional  characteristics.  FBLs  shall 
be  initially  entered  into  the  Government’s  configuration  status  accounting  information  system,  by  means  of  a 
Government-approved  Engineering  Release  Records  (ERR)  system. 


24.  Level  Breakdowns  for  Hardware  and  Software  (HW/SW) 

Various  HW/SW  level  breakdowns  are  developed  at  different  stages  in  the  design  process  to  give  project 
management  personnel  top-down  and  bottom-up  perspectives  on  how  the  system  physically  comes  together. 
Each  level  breakdown  must  reflect  agreement  among  design,  production  and  manufacturing,  software,  and 
logistics  disciplines;  plus  reflect  contract  requirements  and  specifications  as  they  are  understood  by  the 
contractor  and  the  government  at  that  time.  Component  level  breakdowns  serve  as  the  basis  for  writing 
critical  item  development  specifications.  Functional,  allocated,  and  product  baselines  are  generated  to  provide 
inputs  to  the  life  cycle  cost  analysis  and  configuration  control  mechanism. 


25.  Cost  and  Operational  Effectiveness  Analysis 

A  Cost  and  Operational  Effectiveness  Analysis  provides  assistance  in  determining  the  relative  importance  of 
alternatives.  It  does  this  by  providing  a  solid  framework  for  evaluating  the  alternatives,  and  by  highlighting 
the  implications  of  alternative  choices.  In  that  regard,  it  is  essential  to: 

•  Compare  equal-cost  or  equal-effectiveness  alternatives. 

•  Show  the  absolute  values  of  measures  by  making  the  facts  available  and  visible  for  each  alternative. 

•  Never  use  schemes  in  which  several  measures  of  effectiveness  are  weighted  and  combined  into  an 
overall  score. 

•  Use  ratios  only  where  appropriate.  Ratios  may  ignore  sufficiency  and  mask  important  differences. 

•  Point  out  dominance  relationships. 

•  Identify  the  more  effective  alternatives  that  are  roughly  equivalent  in  cost,  and  the  less  costly 
alternatives  that  are  about  equal  in  effectiveness. 

•  For  alternatives  with  comparable  costs  and  effectiveness,  identify  those  that  are  weaker  with  regard 
to  the  more  important  (or  more  frequent)  objectives,  and  those  that  incur  risks  without  producing 
compensating  benefits. 
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•  Highlight  factors  that  may  help  in  ranking  the  remaining  alternatives. 

•  Reexamine  the  base  case  alternative  in  light  of  the  new  insights.  It  may  well  be  better  than  was  first 
perceived,  or  it  may  have  turned  out  to  be  such  a  poor  choice  as  to  make  otherwise  unattractive 
alternatives  quite  appealing. 


26.  Bread  Board  Components  and  Test 

Bread  board  hardware  is  the  first  implementation  of  the  design  concept  in  hardware.  The  intent  of  this  early 
hardware  is  to  prove  the  feasibility  of  the  conceptual  design.  Form  and  fit  considerations  are  not  addressed, 
only  function.  The  components  are  usually  commercial  grade,  or  whatever  is  available  to  perform  the 
function.  The  completed  bread  board  is  tested  in-house  and  at  various  facilities  to  demonstrate  feasibility. 


27.  Life  Cycle  Cost  (LCC)  Analysis 

Life  Cycle  Cost  (LCC)  Analysis  is  a  tool  used  to  incorporate  design,  operation,  and  support  cost  impacts  into 
the  design  process.  In  addition,  all  modernization  activities  will  be  checked  with  LCC. 

LCC  is  also  an  analysis  tool  that  is  used  to  view  a  product  and  its  support  system  from  an  economic 
standpoint  during  the  design  process.  LCC  analyses  address  the  following  items: 

•  Research 

•  Development 

•  Production 

•  Operation 

•  Maintenance 

•  Phase  Out 

•  Modernization 

LCC  tradeoffs  are  traceable  back  to  customer  requirements  via  interplay  between  the  specifications  tree,  level 
breakdown,  and  LCC  analyses. 

The  LCC  process  is  repeated  whenever  any  of  the  factors  affecting  product  design  or  support  change. 


28.  Reviews 

Internal  and  government  required  reviews  are  scheduled  for  hardware  and  software;  government  required 
reviews  are  conducted  per  MIL-STD  1521.  Agendas  include  all  company  and  government  concerns.  A  review 
should  NOT  be  scheduled  before  the  activities  supporting  its  objectives  have  been  completed.  Technical 
issues  are  presented  in  a  manner  that  match  the  audience's  technical  expertise  and  fosters  understanding.  All 
disciplines  that  play  a  role  in  the  transition  to  production  should  have  input  to  the  reviews. 


29.  Request  For  Proposal  (RFP)  and  Source  Selection  Board  (SSEB) 

A  Request  For  Proposals  (RFP)  is  prepared  and  sent  to  industry  so  the  buyer  can  determine  which  company’s 
approach  represents  the  best  value  to  them  (or  with  only  a  single  supplier,  it  details  what  the  buyer  can 
expect).  A  Source  Selection  Evaluation  Board  (SSEB)  is  established  which  develops  a  plan  to  evaluate  the 
proposals  submitted  to  the  buyer(s)  in  response  to  the  RFP. 


30.  Technical  Database 

A  technical  database  is  established  at  the  time  of  contract  award  and  maintained  by  configuration  and  data 
management  personnel.  It  contains  such  information  as  MIL-SPECS,  PIDS  and  Spec  Tree  (and  related 
profiles),  data  deliverables,  program  memos/reports,  information  for  design  reviews  and  PRR.  All  relevant 
items  are  included  in  the  database  when  available  and  updated  when  appropriate.  Database  information 
should  all  be  current  prior  to  each  internal  and  government  required  design  review. 


31.  Post  Award  Conferences 

There  are  two  types  of  post  award  conferences: 

•  A  customer  conference  follows  the  proposed  system  review.  The  purpose  is  to  discuss  technical  and 
funding/provisioning  issues,  schedule  field  visits,  and  agree  on  program  schedule. 

•  A  supplier  (subcontractor)  conference  follows  a  detail  design  review.  The  purpose  is  to  verify  all 
appropriate  contract  requirements,  inform/remind  suppliers  of  incoming  inspection  policy,  introduce 
suppliers  to  individual  points  of  contact  in  the  company,  and  discuss  quick  reaction  capabilities. 


32.  Field  Visits 

Field  visits  are  conducted  to  gain  first  hand  experience  about  how  and  where  the  system  will  be  used  and 
maintained  and  to  identify  operational  constraints.  At  least  one  visit  should  be  conducted;  if  radically 
different  deployment  sites  exist,  each  should  be  visited.  Field  visits  should  occur  early  enough  to  impact 
logistics  profile  requirements  and  PIDS. 


33.  Configuration  Management 

As  customer-approved  baselines  are  established,  the  scope  of  configuration  control  increases.  Upon 
establishment  of  the  product  baseline,  formal  configuration  control,  including  configuration  audits,  requires 
customer  approval.  Prior  to  this,  the  only  changes  requiring  customer  approval  are  those  affecting  approved 
specifications.  The  Configuration  Control  Board  (CCB)  integrates  all  functional  areas  to  effectively  evaluate 
proposed  change  impact  and  approve  or  disapprove  changes. 


34.  Design  Concepts 

Design  concepts  identify  all  reasonable  system  alternatives  (architectures  and  support  methodologies)  that 
may  satisfy  the  mission  need  and  include  recommendations  to  the  program  office.  The  program  manager  then 
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selects  those  alternatives  or  concepts  which  meet  cost,  risk,  schedule,  and  readiness  objectives.  Since  the  use 
of  Computer  Aided  Design  (CAD)  technology  is  a  significant  factor  in  reducing  the  risk  in  development 
projects,  the  use  of  CAD  technology  should  be  encouraged. 

A  design  concept  will  evolve  into  the  design  process  that  includes  continuous  trade-offs. 


35.  Interfaces 

Shortly  after  the  architecture  of  a  product  is  chosen,  the  interfaces  within  and  outside  the  product  must  start  to 
be  finalized  and  put  under  configuration  control. 


36.  Critical  Items  List 

A  critical  items  list  is  developed  and  continuously  maintained.  The  list  must  include:  any  item  of  high 
technical  risk  with  no  work  around,  any  item  with  a  schedule  delivery  risk,  all  sole  source  items,  all 
high-failure-rate  items  (connectors  and  cables  are  automatically  listed).  Critical  Item  Development 
Specifications,  and  all  Safety-of-Flight  material.  Trade  studies  and  lessons  learned  are  incorporated  to  ensure 
all  critical  items  are  identified  at  appropriate  stages  for  consideration  by  program  management. 


37.  Advanced  Development  Model  (ADM)  Fabrication 

Advanced  Development  Model  fabrication  provides  the  first  time  the  product  team  can  determine  what  was 
right  and  what  was  wrong  from  their  modeling  and  simulation  efforts. 


38.  Advanced  Development  Model  (ADM)  Software  Development 

Today,  software  development  represents  an  ever-increasing  part  of  the  development  process.  CASE  tools 
must  be  chosen  and  evaluated  for  the  full  scale  development  process.  Key  Factors  at  this  time  include: 

•  What  exception  conditions,  violation  limits,  and  capacity  limits  have  been  taken  into  account? 

•  What  modifications  are  needed  as  a  result  of  preliminary  prototype  testing? 

•  What  percentage  of  the  software  can  be  reused  from  earlier  systems? 

•  How  many  standard  and  off-the-shelf  software  components  are  estimated  for  the  final  system? 


39.  Advanced  Development  Model  (ADM)  Mock-ups 

Mock-ups  are  an  important  part  of  early  development  to  review  ergonomics  and  equipment  placement. 


40.  ADM  Tests 

Tests  of  an  ADM  have  different  goals  than  tests  later  in  development.  ADM  tests  should  validate  design 
assumptions  and  tools  as  well  as  the  product  itself 
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41.  Refine  Design 

Information  gathered  from  the  ADM  development  and  test  process  is  used  to  further  refine  the  design. 


42.  Human  Factors  Analyses 

Incidental  to  the  ILS  program,  the  Logistics  Analysis  and  manpower  requirements  must  be  determined  early 
in  the  Program  Definition  &  Risk  Reduction  phase.  These  analyses  affect  the  system  design  while  also 
providing  accurate  information  for  use  in  planning  manpower  availabilities  and  skill  levels.  The  analysis 
incorporates  human  engineering  design  criteria,  principles,  and  practices  to  successfully  integrate  the  human 
into  the  system,  subsystem,  equipment,  and  facility.  They  permit  optimum  compatibility  between  human 
performance  and  system  or  equipment  requirements  for  checkout,  operation,  maintenance,  or  control.  The 
results  are  also  helpful  to  establish  user  procedures  and  avoid  excessive,  or  inappropriate  manpower  or  skill 
problems. 


43.  Integrated  Diagnostics/Testability  (ID/T) 

Achievement  of  Diagnostic  and  Testability  features  results  from  early  and  clear  requirements  definition  to  the 
designer.  This  effort  specifically  screens  the  design  and  looks  for  cost-effective  use  of  BIT,  ATE,  and  manual 
testing.  Effective  analyses  will  ensure  that  assembly,  integration,  acceptance,  performance,  diagnostics,  and 
maintenance  testing  are  all  considered  within  this  scope.  This  effort  also  ensures  that  producibility  is  included 
in  the  design  process,  utilizing  proven  manufacturing  processes. 


44.  Draft  Software  User’s  Manual 

Coding  and  related  software  development  activities  should  be  focused  on  top-level  requirements.  Begun  after 
human  factors  analysis,  a  draft  SW  user's  manual  is  completed  prior  to  the  start  of  coding,  deriving  elements 
from  PIDS,  Spec  Tree,  and  system  segment  specifications.  The  manual  provides  other  disciplines  with  insight 
into  software  requirements,  the  efforts  being  made  to  meet  them,  and  how  these  efforts  relate  to  their  own 
activities. 


45.  System  Segment  Specifications 

Just  as  the  Spec  Tree  and  associated  requirements  must  be  developed  down  to  the  lowest  replaceable 
assembly  level,  so  must  the  HAV  and  SAV  breakdowns  flow  continuously  down  to  the  piece-part  or  code 
levels.  This  event  identifies  the  requirement  for  each  segment  specification  to  be  developed  fiilly,  reflect 
agreement  with  other  segment  specifications,  as  well  as  agreement  among  the  various  disciplines  incorporated 
within  the  segment  specification.  This  event  represents  one  formal  stage  in  completing  the  total  system 
design. 


46.  Software  Allocated  Baseline 

The  Software  Allocated  Baseline  is  the  initially  approved  documentation  describing  the  software's  functional, 
interoperability,  and  interface  characteristics.  These  characteristics  are  allocated  from  a  system  or  higher  level 
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configuration  item,  interface  requirements  with  interfacing  configuration  items,  additional  design  constraints, 
and  the  verification  required  to  demonstrate  the  achievement  of  those  specified  characteristics. 


47.  Supplier/Subcontractor  Requirements 

All  specifications,  SOW,  etc.,  for  goods  and  services  are  established  and  documented,  reflecting  all 
requirements  necessarily  imposed  either  by  the  prime  contract  or  company  policy.  One  dedicated 
specifications  group  is  used  to  ensure  completeness  and  consistency  of  specifications,  procurement  packages, 
technical  interfaces  and  imposed  requirements. 


48.  Determine  Specification  Allocations 

A  review  must  be  conducted  to  determine  how  the  design  or  product  will  be  tested  and  proven  to  meet 
customer  needs.  The  goal  is  to  use  as  high  a  level  of  performance  specifications  as  possible  to  allow  the 
contractor  maximum  flexibility  in  achieving  the  desired  goal.  However,  many  times  it  is  hard  to  impossible 
to  test  a  requirement  through  a  performance  test;  these  requirements  should  be  specified  using  a  detailed 
specification.  This  specification  should  call  out  any  critical  processes  that  must  be  followed,  special 
materials  to  be  used,  and  assemble  techniques  needed  to  ensure  product  integrity. 


49.  Manufacturing  Plan 

All  actions  required  to  produce,  test,  and  deliver  acceptable  systems  on  schedule  and  at  minimum  cost  should 
have  been  defined  in  the  manufacturing  plan.  Hence,  the  materials,  fabrication  flow,  time  in  process,  tools, 
test  equipment,  plant  facilities,  and  personnel  skills  are  described  and  integrated  into  a  complete  sequence  and 
schedule  of  events.  It  is  essential  that  both  prime  contractor  activities  and  subcontractor  activities  be  included 
in  the  sequence  and  schedule  of  events. 

Additional  aspects  to  be  incorporated  into  the  Manufacturing  Plan  include  the  extent  of  computer-aided- 
manufacturing,  electro-static  discharge  protection,  environmental  stress  screening,  subcontractor  and  factory 
acceptance  and  qualification  testing,  and  provisions  for  statistical  process  controls.  Early  recognition  and 
selection  from  these  options  will  permit  orderly  assignment  of  responsibilities  and  the  evolution  of  a  best 
manufacturing  strategy,  with  low  risk  and  costs. 


50.  Tool  Plan 

A  tool  plan  identifies  program  tooling  requirements,  including  a  tool  inventory  control  system,  and  integrates 
the  design  and  control  of  tooling  with  the  prime  equipment  development.  Its  objective  is  to  ensure  adequate 
tool  planning  and  control  throughout  the  design  and  production  phases.  The  tool  plan  is  separate  from  the 
manufacturing  plan  but  is  integrated  into  and  updated  with  the  Transition  Plan. 


51.  Transition  Plan 

The  Transition  Plan  (fi*om  development  to  production)  is  developed  as  part  of  the  program  plan  and  serves  as 
a  guideline  to  reflect  an  integrated  corporate  strategy  for  the  program.  It  begins  following  contract  award, 
implemented  during  Engineering  and  Manufacturing  Development  (EMD),  and  is  not  complete  until 
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approved  by  upper  management  and  reviewed  by  the  government.  The  plan  may  use  DoD  manual  4245.7-M 
and  will  contain  mechanisms  to  allow  evaluation  of  the  program's  progress  and  compliance  with  the  plan. 


52.  FaUure  Reporting  Analysis  and  Corrective  Action  System  (FRACAS) 

The  main  objective  of  a  closed-loop  FRACAS  is  to  document  failures,  analyze  their  cause,  determine 
corrective  actions,  and  disseminate  the  data.  Timely  dissemination  of  accurate  failure  information  is  necessary 
for  remedial  actions  to  be  taken  to  prevent  failure  recurrence.  FRACAS  is  complete  only  when  "all  failures 
have  been  identified  and  corrected;"  no  failures  may  be  considered  random,  and  all  failures  are  reported, 
analyzed  and  corrected. 


53.  Long  Lead  Parts/Prove  Out  Process 

While  many  program  managers  understand  the  need  to  order  long  lead  parts  and  materials,  very  few 
understand  the  need  to  prove  out  "long  lead"  processes  so  that  they  will  be  mature  at  production  time. 


54.  Support  Equipment  (SE)  SOW 

The  objective  of  this  activity  is  to  ensure  support  equipment  plarming  and  development  requirements  will  be 
defined  concurrent  with  design  requirements  and  communicated  to  suppliers. 

This  activity  ensures  that  SE  planning  and  development  activities  are  defined  and  communicated  to  suppliers 
together  with  system  requirements.  Additionally,  such  timely  definition  enables  system  design  and  user 
communities  to  reassess  the  design  where  negative  SE  impacts  emerge. 


55.  Special  Test  Equipment  (STE)  SOW 

The  objective  of  this  activity  is  to  ensure  special  test  equipment  planning  and  development  requirements  will 
be  defined  concurrent  with  design  requirements  and  commimicated  to  suppliers. 

This  activity  ensures  that  STE  planning  and  development  activities  are  defined  and  communicated  to 
suppliers  together  with  system  requirements.  Additionally,  such  timely  definition  enables  system  design  and 
user  communities  to  reassess  the  design  where  negative  STE  impacts  emerge. 


56.  Manufacturing  Process  Development  Plan 

The  goal  of  this  plan  is  to  determine  the  best  manufacturing  strategy  (i.e.,  one  that  provides  low 
manufacturing  risk  and  lower  costs)  by  achieving  a  balance  between  proven  manufacturing  processes  and 
newer  techniques.  Critical  manufacturing  processes  are  identified  and  agreed  upon  by  engineering  and 
manufacturing  personnel  who  are  ideally  collocated  during  development.  No  manufacturing  should  take  place 
until  processes  developed  for  the  program  are  mature  enough  to  support  it. 
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The  following  major  items  are  considered  in  support  of  the  Manufacturing  Process  Development  Plan: 

•  Schedule 

•  Budget 

•  Hardware  Utilization  Mix 

•  Capital  Procurement 

•  Risk  Management  Plan 

•  Manufacturing  Plan/Strategy 

•  Make/Buy  Plan  (Including  updates) 

•  Transition  to  Production  Plan  (Including  updates) 

•  Manufacturing  Technology  Capabilities  Assessment 

•  Subcontract  Management  Plan 

•  Test  Equipment  Approach/Goals 

•  Major  Subcontractor  Selection 


57.  Technical  Manuals 

A  complete  set  of  technical  manuals  is  developed  concurrent  with  HW/SW  development.  Ideally, 
manufacturing  process  development  is  integrated  with  technical  manual  development.  Responsibilities  for 
technical  manuals  are  outlined  in  the  Integrated  Logistics  Support  (ILS)  plan  and  maintenance  tasks  are 
identified  via  Logistics  Analysis.  Outlines  are  submitted  to  the  government  for  review  and  comments  prior  to 
final  publication;  interim  versions  are  available  for  training  course  development. 


58.  Software  Simulator 

A  simulator  is  built,  used,  and  maintained  throughout  the  program’s  life  for  software  simulation  testing.  A 
software  top-level  design  document  is  completed  prior  to,  and  includes  a  plan  for  the  simulator  build.  The 
simulator  is  completed  before  software  simulation  testing  can  begin. 

Software  simulators  should  be  considered  and  requirements  developed  during  the  planning  and  requirements 
phase  to  effect  a  significant  item  of  risk  reduction  during  test  implementation,  especially  for  large  systems. 
This  will  ensure  that  applicable  metrics,  standards,  and  responsibilities  for  their  implementation  are  assigned 
in  a  timely  manner.  Simulators  also  provide  potential  benefits  by  defining  integration  issues,  and  can  identify 
the  probability  and  impact  of  imwanted  latent  paths  or  timing  which  may  be  considered  for  sneak  circuit 
analysis. 


59.  Engineering  Drawings  (Level  I  &  II) 

A  complete  technical  documentation  package  may  be  a  contract  deliverable  for  hardware  to  be  purchased  by 
the  government.  It  allows  any  qualified  manufacturer  to  build  the  hardware.  Development  of  product 
drawings  and  associated  lists  begins  after: 

•  Conceptual  and  developmental  design  drawings  and  associated  lists  have  completed  detail  design 
reviews. 

•  The  critical  design  review  is  completed. 
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60.  Establish  Diminishing  Manufacturing  Sources  (DMS)  Mitigation  Working  Group 

A  top  problem  today  is  parts  obsolescence.  Currently  the  U.S.  parts  base  is  turning  over  in  as  little  as  24 
months.  This  means  that  if  you  take  more  then  24  months  from  CDR  to  production  you  may  have  to  redesign 
just  due  to  parts  obsolescence  problems.  There  are  many  ways  to  deal  with  this  problem;  two  critical 
activities  are  the  establishment  of  a  parts  tracking  database  and  rules  for  the  rapid  insertion  of  new 
technologies  (e.g.,  what  level  of  regression  testing).  An  important  task  of  this  working  group  between  the 
PDR  and  CDR  is  to  review  the  design  for  vulnerability  to  parts  obsolescence.  Certain  design  philosophies 
(e.g.,  use  of  FPGAs  instead  of  ASICs)  are  much  less  susceptible  to  parts  obsolescence.  Associated  issues 
include  parts  derating,  100%  parts  re-screening  of  commercial  and  plastics  parts  for  the  military  environment, 
and  Environmental  Stress  Screening  (ESS). 


61.  Technology  Maturation  Loop 

While  most  acquisition  and  development  time  lines  are  drawn  as  linear  events  progressing  from  left  to  right, 
in  reality  this  is  not  how  the  process  actually  works.  At  a  high  level  the  time  lines  look  serial  or  sequential, 
but  actually  have  two  “loops”  where  traditional  program  management  tools  and  techniques  are  not  effective. 
The  first  of  these  we  call  the  Technology  Maturation  Loop.  During  this  time  a  series  of  some  10  to  12 
hardware  and  software  events  are  repeated  in  an  iterative  manner  to  mature  the  design.  Many  factors  affect 
the  length  of  time  spent  in  this  loop  including  technology  maturity,  contractor  experience,  and  design  analysis 
tools  used.  During  these  maturation  loops  it  is  EXTREMELY  hard  if  not  impossible  to  predict  schedule  with 
high  reliability.  See  discussion  at  end  of  Section  Four  entitled  ” Accurate  Schedule  Reporting  Through 
Tiered  Milestone  Dating  ” 


62.  Training 

Training  consists  of  the  processes,  procedures,  techniques,  training  devices,  and  equipment  used  to  train 
civilian,  active  duty,  and  reserve  military  personnel  to  operate  and  support  equipment.  A  training  plan  should 
be  developed  to  identify  a  schedule  for  the  life  cycle  manpower,  personnel,  and  training  requirements 
associated  with  planning  the  introduction  of  new  equipment,  systems,  or  subsystems.  This  includes 
equipment  maintenance  and  manning  requirements,  a  training  concept  for  instructors,  individual  and  crew 
training  (both  initial  and  continuation),  and  on-the-job  training.  Training  shall  also  address  logistics  support 
planning  for  the  training  site,  curriculum,  training  materials,  technical  training  equipment,  technical  manuals, 
special  tools  and  test  equipment,  spare  parts,  training  device  acquisitions  and  installations. 


63.  Incoming  Inspection 

An  effective  parts  control  program  requires  that  feedback  techniques  be  established  which  notify  suppliers  of 
defective  parts  and  require  corrective  action  on  the  part  of  the  vendor.  In  addition,  visibility  of  incoming  and 
assembly  yield  rates  must  be  continuously  maintained  to  identify  poor  suppliers  and  to  develop  more  effective 
incoming  and  vendor  screens.  Early  detection  of  parts  problems  is  the  key  to  a  low  risk  transition  to  rate 
production. 
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64.  Establish  Pedigree  Process 

Today  due  to  the  heavy  use  of  statistics  (6  Sigma,  etc.)  in  the  design  and  production  process  the  ability  to 
build  one  high  quality  prototype  (or  Golden  Round)  has  been  lost.  A  key  part  of  this  process  is  documenting 
the  pedigree  of  all  parts  and  assemblies.  This  detailed  documentation  and  tracking  of  all  parts  and  assemblies 
allows  test  engineering  to  know  (or  know  they  don’t  know)  the  quality  of  the  Unit  Under  Test  (UUT).  This 
process  also  provides  feedback  to  the  assemblers  at  times  when  (for  example  a  lot  of  squibs  showed  40% 
failure  rate)  you  don’t  want  to  use  an  item  from  this  lot  in  your  test  round.  Without  a  strong  pedigree  process 
you  will  not  even  know  if  your  parts  came  from  this  lot!  A  good  pedigree  process  insures  traceability  and 
documentation  at  all  levels  covering  the  four  main  categories  of  MAN,  METHODS,  MACHINE,  and 
MATERIALS. 


65.  Engineering  Design  Model  (EDM) 

Because  the  design  process  involves  all  the  disciplines  of  the  integrated  product  team,  the  use  of  uniform  and 
standard  practices  improves  the  resulting  schedule  realism  and  accuracy  of  the  design.  Construction  of  the 
Engineering  Design  Model  (EDM)  is  one  such  formal  stage  in  validating  the  design  with  manufacturing 
disciplines.  The  EDM  serves  to  proof  the  design  on  manufacturing  models  with  feedback  of  the  results  to 
engineering.  This  step  augments  similar  engineering  information  flow  from  design  review  findings. 
Sometimes  this  run  of  imits  is  extended  prior  to  a  production  award  into  prototypes  and  a  Low  Rate  Initial 
Production  (LRIP). 


66.  Production  Readiness  Review  (PRR) 

Production  Readiness  Review  (PRR)  is  intended  to  determine  the  completion  status  of  specific  activities 
required  for  production  go-ahead  decisions.  PRR  occurs  incrementally  during  EMD  addressing  all  areas  of 
concern  in  the  manufacturing  plan.  Early  stages  are  devoted  to  gross  level  manufacturing  concerns  and 
progress  to  a  more  detailed  level  as  the  design  matures.  There  is  no  upper  limit  to  the  number  of  incremental 
PRRs;  PRR  schedules  relate  to  major  design  milestones,  but  are  not  specifically  related  to  other  design 
reviews.  Subcontractors  and  suppliers  are  included  in  PRRs.  A  PRR  is  complete  when  both  the  contractor  and 
government  have  verified  that  all  activities  required  to  support  a  production  go-ahead  decision  have  been 
completed. 


67.  Interim  Contractor  Support 

When  funded  by  the  government,  interim  contractor  support  is  intended  to  ensure  equipment  supportability 
and  continued  FRACAS  activity  until  the  prime  equipment  and  its  spares  are  fully  integrated  into  the 
government  inventory.  Contractor  engineering  teams  provide  independent  engineering  analysis  of  field  data 
and  are  required  to  provide  details  of  all  failures. 


68.  Provisioning  and  Field  Support 

While  numerous  logistics  studies  and  analysis  many  have  been  conducted,  provisioning  is  the  first  step  to 
supporting  an  operational  system.  Very  few  systems  today  are  "throw  away"  so  field  support  is  important. 
Field  support  should  not  only  be  used  to  ensure  customer  satisfaction  but  also  as  feedback  to  the  design  and 
production  process. 
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69.  Statistical  Process  Control  (SPC) 

Statistical  Process  Control  (SPC)  takes  on  many  forms.  It  is  important  that  it  be  tailored  to  the  particular 
product  in  question.  Use  SPC  for  process  monitoring,  problem  solving,  and  communication.  Teach  basic 
SPC  techniques  to  all:  check  sheets,  histograms,  Pareto  analysis,  cause  and  effect  analysis,  and  control  charts. 
Teach  advanced  SPC  techniques  to  engineers  and  operators:  multivariate  charts,  design  of  experiments,  and 
capability  charts. 


70.  Factory  Acceptance  Tests 

Once  the  design  has  matured,  Factory  Acceptance  Tests  are  developed  to  ensure  that  each  unit  performs  as 
the  test  units  did. 


71.  Physical  Configuration  Audit  (PCA)/Functional  Configuration  Audit  (FCA) 

These  audits  will  verify  that  configuration  items  have  been  developed  satisfactorily  and  that  they  achieve  the 
performance  and  specified  functional  characteristics.  In  addition,  configuration  items  will  be  technically 
examined  to  assure  that  the  ”As  Built”  configuration  items  conform  to  the  technical  documentation  that 
defines  them.  The  design  package  cannot  be  released  to  production  until  the  audits  have  been  successfully 
conducted. 


72.  Process  Maturation  Loop 

While  most  acquisition  and  development  time  lines  are  drawn  as  liner  events  progressing  from  left  to  right  in 
time,  in  reality  this  is  not  how  the  process  actually  works.  The  high  level  process  looks  serial  but  actually  has 
two  “loops”  where  traditional  program  management  tools  and  techniques  are  not  effective.  This  “loop”  is 
similar  to  the  Technology  Maturtion  Loop  discussed  during  the  design  phase,  except  now  instead  of  maturing 
the  design  it  is  the  production  process  that  matures  or  “stabilizes”  resulting  in  low  variability.  During  this 
time  a  series  of  approximatelly  12  events  are  repeated  in  an  iterative  manner  to  mature  the  process.  This  is 
the  second  place  where  traditional  program  management  tools  and  techniques  are  not  effective.  During  these 
maturation  loops  it  is  EXTREMELY  hard  if  not  impossible  to  predict  schedule  with  high  reliability.  See 
discussion  at  end  of  Section  Four  entitled  ''Accurate  Schedule  Reporting  Through  Tiered  Milestone 


73.  First  Article  &  Qualification  Test  (FAQT) 

These  tests  address  environmental  and  life  test  requirements,  as  documented  in  PIDS  and  the  integrated  test 
plan.  The  purpose  is  to  measure  actual  hardware  and  software  configuration  items  performance  for 
compliance  with  PIDS,  software  requirements,  and  interface  requirements  specifications.  Testing  occurs  on 
preproduction  hardware.  Test  results  are  documented  in  a  format  that  facilitates  government  interpretation 
and  certification. 


74.  Test,  Analyze,  and  Fix  (TAAF) 

Test,  Analyze,  and  Fix  (TAAF)  is  a  planned  process  in  which  development  items  are  tested  under  actual  or 
simulated  mission  profile  environments  to  disclose  design  deficiencies  and  to  provide  engineering  information 
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on  failure  modes  and  mechanisms.  The  purpose  of  TAAF  is  to  provide  a  basis  for  early  incorporation  of 
corrective  actions  and  verification  of  their  effectiveness  in  improving  equipment  reliability. 


75.  Factory  Layout  and  Flow 

Factory  improvement  does  not  just  mean  updating  the  equipment  in  the  factory.  It  involves  striving  to  achieve 
an  integrated  and  supportive  mix  of  layout,  material  flow,  inventory  control  and  flow,  manufacturing 
processes,  maintenance,  and  plant  control.  An  organization  must  be  able  to  assess  the  value-added  of  the 
technology  before  investing  capital. 


76.  DMS  Parts  Tracking 

Identification  and  prevention  of  parts  obsolescence  requires  a  dynamic,  proactive  process.  Examination  of 
the  design  prior  to  the  Production  Readiness  Review  provides  a  timely  opportunity  to  ensure  that  the 
design  reflects  parts  and  materials  that  will  be  available  throughout  production  and  supportable  in  use.  At 
this  point,  parts  and  materials  at  risk  of  obsolescence  need  to  be  identified  and  replaced.  DMS  is  a 
producibility  issue  and,  therefore,  a  design  term  rather  than  a  problem  that  simply  materializes  during 
production  and  support. 


77.  Build  Systems 

The  actual  building  of  the  system  is  what  all  the  design  efforts  were  about.  The  goal  now  is  to  meet 
production  schedules,  constantly  reduce  costs,  cycle  time,  and  variability. 


78.  Test  Equipment  Certification/Calibration 

It  is  important  that  test  equipment  have  the  accuracy,  precision,  and  repeatability  to  measure  the  critical 
aspects  of  the  product  and  its  performance. 


79.  Production  Worker  Training 

Training  programs  should  be  in  effect  which  satisfy  current  and  future  needs  of  both  employer  and  employee. 
Personnel  training  is  a  critical  ingredient  in  ensuring  the  stability  of  the  manufacturing  operation,  A  "hands 
on"  training  program  should  be  in  place  for  factory  personnel.  This  training  should  be  accomplished 
"off-line"  using  the  same  equipment,  procedures,  and  work  instructions  that  will  be  required  on  the  factory 
floor.  Work  force  stability  should  be  tracked.  Personnel  turnover  and  level,  and  quantity  and  level  of  training 
should  be  readily  accessible  and  tracked.  The  introduction  of  new  equipment  or  personnel  on  the  product  line 
without  successful  completion  of  "hands  on"  training  should  never  occur. 


80.  Develop  and  Implement  Engineering  Change  Proposals  (ECPs)  as  Required 

The  need  for  an  Engineering  Change  Proposal  (ECP)  occurs  whenever  there  is  a  problem  or  an  opportunity 
for  improvement.  An  ECP  generates  an  alteration  to  a  product's  attributes  or  its  associated  design.  An  ECP 
can  also  be  used  to  change  the  released  configuration  documentation  of  a  product.  Effecting  an  engineering 
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change  may  involve  modification  of  the  product,  product  information  and  associated  interfacing  items. 
Changes  to  released  documentation  must  be  evaluated  for  the  effect  on  users  of  the  released  information  who 
may  have  begun  ordering  material,  designing  tooling,  planning  manufacturing,  coding  software  or  producing 
parts.  ECPs  are  classified  as  Class  I  or  Class  II  depending  on  the  extent  of  the  impact. 


81.  Production  Breaks 

Production  breaks  can  unavoidably  be  caused  by  project  justification  delays  or  funding  problems.  This 
normally  results  in  either  a  slowdown  or  a  temporary  shutdown  of  production.  Production  delays  always 
result  in  overall  project  cost  increases.  The  primary  reasons  production  breaks  increase  costs  are: 

•  Skilled  employees  must  be  paid  from  funded  projects  or  be  released  from  employment  in  order  for 
any  company  to  remain  profitable. 

•  The  additional  costs  to  train  replacement  workers. 

•  Inflation. 

•  Job  instability  affects  personal  security  and  motivation. 

•  Floor  space  and  expensive  machinery  are  valuable  resources  that  cannot  remain  idle. 

•  Loss  of  established  vendors. 

•  A  production  restart  effort  means  the  lessons  learned  originally  must  be  learned  again. 

•  Ultimately,  the  overall  cost  to  produce  each  unit  increase. 
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SECTION  FIVE 


ACCURATE  PROGRAM  SCHEDULE  REPORTING 
THROUGH  TIERED  MILESTONE  DATING 


A  major  problem  facing  program  managers  today  is  meeting  schedules.  Developing  and  maintaining 
schedules  are  very  important  aspects  of  an  acquisition  program.  So  why  do  military  acquisition  programs 
have  such  difficulty  in  meeting  schedules?  To  answer  this,  we  must  realize  that  the  problem  is  multi-faceted. 
DoD  and  GAO  have  identified  a  variety  of  contributors.  One  key  element  is  the  lack  of  validated  models  to 
predict  future  events. 

Schedule  overruns  seem  to  moimt  on  top  of  previous  schedule  overruns,  and  soon  senior  decision  makers 
have  no  faith  in  the  promises  of  their  program  managers.  Since  schedules  are  actually  predictions,  then  we 
must  look  at  the  methods  we  use  to  predict  future  events. 

The  ability  to  accurately  predict  an  outcome  is  based  on  two  factors:  (1)  an  understanding  of  the  process,  and 
(2)  the  ability  to  consistently  follow  this  process.  The  Navy’s  Best  Manufacturing  Practices  Center  of 
Excellence  (BMPCOE)  has  been  developing  and  maintaining  a  systems  engineering  model,  and  developing 
process  control  tools  (e.g.,  the  Technical  Risk  Identification  and  Mitigation  System  [TRIMS])  for  some  time. 
While  these  tools  have  proven  to  be  effective  for  instilling  repeatability  into  the  systems  engineering  process, 
schedule  overrun  problems  still  persist. 


When  the  BMPCOE  started  out  to  study  schedule  overrun  problems,  we  first  reviewed  the  acquisition  process 
being  used.  In  this  case,  the  baseline  is  the  DoD  5000  acquisition  process.  Figure  2  shows  an  overview  of  a 
basic  DoD  acquisition  timeline. 


Figure  2  -  A  Linear  Assumption  of  the  Acquisition  Process 


This  acquisition  process,  as  published,  is  basically  linear  and  divided  into  phases  that  depict  the  maturation  of 
a  system  as  time  progresses.  While  there  are  many  iterative  processes  involved,  the  basic  process  at  the  top 
level  is  still  depicted  (and  managed)  as  though  it  were  linear.  To  further  illustrate  this  belief  is  the  fact  that 
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we  typically  publish  an  IOC  for  a  program  almost  at  Day  1 !  Herein  is  where  the  problem  lies:  the  process  at 
the  top  level  is  not  linear. 

There  are  two  critical  points  that  occur  during  a  DoD  acquisition  program’s  life.  These  points  are  best 
described  as  maturation  loops  as  shown  in  Figure  3.  The  loops  represent:  (1)  the  design 
maturation  process  where  modeling  and  simulation  tools  do  not  exist,  are  poorly  characterized,  or  were  not 
used;  and  (2)  the  production  maturation  process  where  the  design  differs  from  the  norm  and  processes  and 
tooling  must  be  tailored.  The  activities  within  these  loops  are  NOT  manageable  with  traditional  program 
management  disciplines  and  tools.  In  fact,  they  exist  because  modeling  is  not  possible  for  the  potentially 
millions  of  interactions  that  are  taking  place.  These  two  breakpoints  render  it  is  almost  impossible  to 
accurately  predict  an  IOC  early  in  a  state-of-the-art  development  program. 


The  comment  has  been  made  “why  don’t  we  see  these  loops  in  commercial  programs”?  In  point  of  fact  we 
do.  Many  purely  commercial  programs  have  built  dozens  of  prototypes  to  get  a  design  correct.  The  lack  of 
these  maturation  loops  in  a  commercial  program  is  typically  because  they  are  not  pushing  state-of-the-art 
concepts.  Most  commercial  programs,  while  producing  a  new  product,  are  based  on  very  mature  technology. 

Consequently,  two  very  different  processes  and  tools  are  needed  to  manage  a  DoD  acquisition  program. 
First,  the  typical  program  management  processes  and  tools  are  needed;  second,  a  method  for  managing 
activities  within  the  maturation  loops  must  be  established.  The  typical  program  management  tools  and 
techniques  are  well  known,  so  we  will  focus  on  the  maturation  loops. 

Since  we  are  maturing  technology,  the  following  adage  is  the  governing  principle: 

‘‘You  cannot  mandate  a  technological  breakthrough.” 
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This  statement  is  not  to  be  taken  lightly.  Most  everyone,  when  asked  if  they  agree  with  this  statement,  will 
publicly  state  that  they  do.  Yet  technological  breakthroughs  are  exactly  what  we  expect  during  these 
maturation  loops,  and  we  expect  them  on  schedule!  These  two  desires  are  diametrically  opposed,  hence, 
program  managers  are  constantly  overrunning  their  predicted  schedules. 

So  what  does  the  process  really  look  like  and  how  should  it  work?  The  process,  shown  in  Figure  3,  is 
representative  of  how  the  acquisition  process  really  works,  and  is  based  on  what  we  call  relative  (or  tiered) 
milestone  dating.  Even  though  the  tools  and  techniques  do  not  exist  to  accurately  predict  when  a  program 
will  exit  a  maturation  loop,  the  periods  before,  between,  and  after  these  loops  are  VERY  predictable  based  on 
proven  program  management  methods.  It  is  in  those  periods  that  a  program  manager  should  be  held 
responsible  for  maintaining  a  program  within  schedule.  Therefore,  an  IOC  should  not  be  published  as  a 
distinct  date  but  rather  as  an  equation  (e.g..  Early  in  a  program  I  cannot  predict  exactly  when  I  will  pass  MSS, 
but  I  do  know  that  it  will  be  X  days  after  a  distinct  technical  event  is  achieved.). 

Inside  the  maturation  loops,  the  process  is  quite  simple  (e.g.,  How  much  money  do  you  want  to  spend  and 
how  fast?).  Since  we  cannot  mandate  a  technological  breakthrough,  what  can  we  do  instead? 

A.  Bring  ALL  design  analysis  tools  to  bear. 

B.  While  exit  points  cannot  be  determined  by  date,  specific  technical  exit  criteria  should  be  established. 

C.  While  exact  exit  dates  cannot  be  determined  at  entry,  increased  fimding  does  typically  shorten  the 
“loop”  duration. 

There  is  a  second  issue  with  regard  to  accurately  predicting  schedules  and  that  is  time  itself.  To  use  an 
example  Irom  aviation  lets  look  at  the  concept  of  navigation  using  an  Inertial  Navigation  System  (INS).  This 
system  keeps  track  of  where  the  aircraft  is  by  keeping  track  of  each  and  every  turn  and  change  in  speed. 
Consequently,  with  no  external  inputs  it  can  tell  you  where  you  are  as  long  as  it  knows  where  you  started  (you 
can  equate  good  requirements,  or  more  importantly  an  allocated  baseline  to  the  starting  point).  But  there  is  a 
problem:  no  machine  is  perfect,  and  as  the  aircraft  travels  along  over  time  very  slight  errors  are  introduced 
into  the  calculation  of  the  aircraft’s  position.  On  a  short  flight  this  is  not  a  big  problem,  your  actual  position 
will  not  vary  much  from  the  displayed  position.  But  on  longer  flights  these  small  errors  accumulate  and  build 
over  time  until  the  aircraft  can  actually  be  many  miles  from  the  displayed  position. 

This  same  process  applies  to  acquisition  programs  and  their  schedules.  The  further  out  one  goes  on  a 
predicated  schedule  the  less  and  less  accurate  it  becomes.  It  is  rarely  possible  to  develop  accurate  plans  more 
then  one  or  two  major  steps  ahead  in  an  acquisition  process  (typically  18  months  to  a  maximum  of  2  years); 
beyond  this  point  plans  are  vague,  non-specific  and  hence  schedules  are  nothing  more  then  guesses.  In  fact 
this  desire  to  predict  IOC  on  day  one,  and  the  subsequent  attempts  to  meet  it  (based  on  vague  planning  and 
lots  of  technical  unknowns)  actually  lengthen  schedules! 
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In  conclusion, 

Tiered  Milestone  Dating 

•  DoD  acquisition  process  is  not  linear  as  published 

•  You  cannot  mandate  technological  breakthroughs:  Different  skills  and 
metrics  needed  to  “manage”  R&D  and  Production  Maturation  Loops 

•  We  must  define  and  understand  entry  and  exit  criteria  for  “Loops” 

•  Strong  use  of  modeling  and  simulation  can  shorten  “Loops” 

3 

•  “Loops”  shortened  by  use  of  “Evolved  Capability”  and  true  P  I 

•  Digital  systems  design  approach  (hardware  and  software  not  separate) 

•  Schedules  depicting  events  more  than  18-24  months  in  advance  are 
merely  guesses 


Results:  More  accurate  reporting 
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